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Program Description 1 

Career Academies were developed more than 30 years ago as a 
dropout prevention strategy for youth considered most at risk of 
dropping out of high school. Students in Career Academies take both 
career-related and academic courses and acquire work experience 
through partnerships with local employers. Since their inception, 

Career Academies have broadened the kinds of students they serve, 
integrated rigorous academic curricula with career themes, and now 
attract students who are preparing for postsecondary education. 

Research 2 

The What Works Clearinghouse (WWC) identified one study of Career 
Academies that falls within the scope of the Dropout Prevention topic 
area and meets WWC group design standards. This study meets 
WWC group design standards without reservations. The study 
included between 1,379 and 1,454 students (depending on outcome) 
who applied to an academy before their ninth- or tenth-grade years. 

Academies were located in eight urban areas in six states. 

The WWC considers the extent of evidence for Career Academies on 
the educational attainment of high-school aged youth to be small for 
three outcome domains— completing school, staying in school, and 
progressing in school. (See the Effectiveness Summary on p. 4 for more 

Effectiveness 

Career Academies were found to have potentially positive effects on completing school and no discernible effects 
on staying in school or progressing in school for high-school aged youth. 

Table 1. Summary of findings 3 


Improvement index 

(percentile points) 


Outcome domain 

Rating of effectiveness 

Average 

Range 

Number of 
studies 

Number of 
students 

Extent of 
evidence 

Completing school 

Potentially positive effects 

+11 

na 

1 

1,428 

Small 

Staying in school 

No discernible effects 

+6 

na 

1 

1,454 

Smal 

Progressing in school 

No discernible effects 

+4 

na 

1 

1,379 

Small 


na = not applicable 
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Program Information 

Background 

Career Academies were first implemented in Philadelphia in 1969 and replicated in California beginning in the 1980s. 
Today, academies are in districts throughout the country, including more than 1 ,200 career academies in nearly 
500 California high schools. Many Career Academies received support from the federal Small Learning Communities 
discretionary grants authorized by the Elementary and Secondary Education Act, and from grants funded by the 
Perkins Career and Technical Education Act. Information on the history of Career Academies and current resources 
for program implementation are available from the National Career Academy Coalition (http://www.ncacinc.com) and 
the Career Academy Support Network (http://casn.berkeley.edu). 

Program details 

This report focuses on Career Academies with a school-within-a-school structure. Each academy has a career 
theme, such as health care, finance, technology, communications, and public service. Students take their career- 
related courses in the academy, which often are taught by a core team of academy teachers. Generally, students 
remain with the same group of students and teachers over time, take both academic and career-oriented courses, 
and participate in work-based learning activities inside and outside of school. National standards of practice 
for Career Academies, created by the National Career Academy Coalition, indicate that three basic features are 
key elements of a career academy. First, Career Academies are small learning communities in which clusters 
of students share several classes each year and teachers collaborate around student needs. Second, Career 
Academies have a focused curriculum with a career theme relevant to local industry and economic needs. 

Third, Career Academies develop partnerships with employers, higher education institutions, and the community. 
Participants in these partnerships advise on curriculum related to occupations, speak in classes, host field trips, 
provide financial or other support, and serve as student mentors. 


Cost 4 

The cost of Career Academies was estimated in 2004 to be $600 per pupil more than a district’s average per-pupil 
expenditure (cost data refer to the California Partnership Academies). More recent cost information was not available 
at the time of this report. 
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Research Summary 

The WWC identified nine eligible studies that investigated the effects Table 2. Scope of reviewed research 
of Career Academies on the educational outcomes of youth at risk of 
dropping out. An additional seven studies were identified but do not 
meet WWC eligibility criteria for review in this topic area. Citations for 
all 16 studies are in the References section, which begins on p. 6. 

The WWC reviewed nine eligible studies against group design standards. 

One study (Kemple & Snipes, 2000) is a randomized controlled trial that meets WWC group design standards without 
reservations. The study is summarized in this report. Eight studies do not meet WWC group design standards. 

Summary of study meeting WWC group design standards without reservations 

Kemple and Snipes (2000) conducted a randomized controlled trial that included between 1 ,379 and 1 ,454 youth 
(depending on outcome) who applied to Career Academies starting at grade nine and who participated in the 
evaluation. The study measured outcomes at the end of a student’s projected twelfth-grade year (Kemple & Snipes, 
2000), 4 years after a student’s projected twelfth-grade year (Kemple, 2004), and 8 years after a student’s projected 
twelfth-grade year (Kemple & Willner, 2008). 

Summary of studies meeting WWC group design standards with reservations 

No studies of Career Academies met WWC group design standards with reservations. 


Grade 

9-12 

Delivery method 

Whole school 

Program type 

Curriculum 
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Effectiveness Summary 

The WWC review of Career Academies for the Dropout Prevention topic area includes outcomes in three domains: 
completing school, staying in school, and progressing in school. The one study of Career Academies that meets 
WWC group design standards reported findings in all three domains. The findings below present the authors’ 
estimates and WWC-calculated estimates of the size and statistical significance of the effects of Career Academies 
on high-school aged youth. For a more detailed description of the rating of effectiveness and extent of evidence 
criteria, see the WWC Rating Criteria on p. 16. 

Summary of effectiveness for the completing school domain 

One study that meets WWC group design standards without reservations reported findings in the completing 
school domain. 

Kemple and Willner (2008) reported that students who enrolled in Career Academies were more likely to earn a high 
school diploma or General Educational Development (GED) certificate of high school equivalency within 8 years 
of their expected high school graduation. About 96% of the students enrolled in Career Academies earned a high 
school diploma or GED certificate within 8 years of their expected high school graduation date, compared to about 
94% of students in the comparison group. The effect size of 0.27 was not statistically significant but was substan- 
tively important. The WWC characterizes these study findings as having a potentially positive effect. 

Thus, for the completing school domain, the one study that met WWC group design standards showed positive 
effects that were not statistically significant. This results in a rating of potentially positive effects, with a small 
extent of evidence. 


Table 3. Rating of effectiveness and extent of evidence for the completing school domain 


Rating of effectiveness 

Criteria met 

Potentially positive effects 

Evidence of a positive effect with 
no overriding contrary evidence. 

In the one study that reported findings, the estimated impact of the intervention on outcomes in the completing 
school domain was not statistically significant but was substantively important. 

Extent of evidence 

Criteria met 

Small 

One study that included 1 ,428 students in nine schools reported evidence of effectiveness in the completing 
school domain. 


Summary of effectiveness for staying in school domain 

One study that meets WWC group design standards without reservations reported findings in the staying in 
school domain. 

Kemple and Snipes (2000) reported that students enrolled in Career Academies were no more likely than similar 
comparison group students to have dropped out of school by the end of twelfth grade. About 10% of the students 
enrolled in Career Academies had dropped out of high school before the end of grade 12, compared to about 12% 
of students in the comparison group. The effect size of 0.14 was not statistically significant or substantively important. 
The WWC characterizes these study findings as an indeterminate effect. 

Thus, for the staying in school domain, the one study that met WWC group design standards did not show either 
a statistically significant effect or an effect large enough to be considered substantively important. This results in 
a rating of no discernible effects, with a small extent of evidence. 
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Table 4. Rating of effectiveness and extent of evidence for the staying in school domain 


Rating of effectiveness 

Criteria met 

No discernible effects 

None of the studies show 
statistically significant or 
substantively important effects, 
either positive or negative. 

In the one study that reported findings, the estimated impact of the intervention on outcomes in the 
staying in school domain was not statistically significant or substantively important. 

Extent of evidence 

Criteria met 

Small 

One study that included 1 ,454 students in nine schools reported evidence of effectiveness in the 
staying in school domain. 


Summary of effectiveness for the progressing in school domain 

One study that meets WWC group design standards without reservations reported findings in the progressing in 
school domain. 

Kemple and Snipes (2000) reported that students enrolled in Career Academies accrued no more credits than 
similar comparison group students by the end of the twelfth-grade year. The WWC-calculated effect size of 
0.09 was not statistically significant or substantively important. The WWC characterizes these study findings 
as an indeterminate effect. 

Thus, for the progressing in school domain, the one study that met WWC group design standards did not 
show either a statistically significant effect or an effect large enough to be considered substantively important. 
This results in a rating of no discernible effects, with a small extent of evidence. 


Table 5. Rating of effectiveness and extent of evidence for the progressing in school domain 


Rating of effectiveness 

Criteria met 

No discernible effects 

None of the studies show 
statistically significant or 
substantively important effects, 
either positive or negative. 

In the one study that reported findings, the estimated impact of the intervention on outcomes in the 
progressing in school domain was not statistically significant or substantively important. 

Extent of evidence 

Criteria met 

Small 

One study that included 1 ,379 students in nine schools reported evidence of effectiveness in the 
progressing in school domain. 
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Appendix A: Research details for Kemple & Snipes, 2000 

Kemple, J. J., & Snipes, J. C. (2000). Career Academies: Impacts on students’ engagement and 
performance in high school. New York, NY: MDRC. 

Additional sources: 

Kemple, J. J. (1997). Career Academies communities of support for students and teachers: Emerging findings 
from a 10-site evaluation. New York, NY: MDRC. 

Kemple, J. J. (2001). Career Academies: Impacts on students’ initial transitions to postsecondary education 
and employment. New York, NY: MDRC. 

Kemple, J. J. (2004). Career Academies: Impacts on labor market outcomes and educational attainment. 
New York, NY: MDRC. 

Kemple, J. J., & Rock, J. L. (1996). Career Academies: Early implementation lessons from a 10-site evaluation. 
New York, NY: MDRC. 

Kemple, J. J., & Willner, C. J. (2008). Career Academies long-term impacts on labor market outcomes, educational 
attainment, and transitions to adulthood. New York, NY: MDRC. 

Table A. Summary of findings Meets WWC group design standards without reservations 


Study findings 
Average improvement index 


Outcome domain 

Sample size 

(percentile points) 

Statistically significant 

Completing school 

1,428 students 

+11 

No 

Staying in school 

1,454 students 

+6 

No 

Progressing in school 

1,379 students 

+4 

No 


Setting Ten sites were chosen for the evaluation because they were implementing components of Career 
Academies (namely, a school-within-a-school organization, academic and vocational curricula 
based on a career theme, and employer partnerships— features not available in participating 
high schools) but were not in the early stages of implementation. Of the 10 sites initially in the 
study, one dropped out after randomization. Nine sites contributed outcome data: three sites 
in the East (the District of Columbia, Maryland, and Pennsylvania), two sites in Florida, one site 
in Texas, and three sites in California. 

Study sample Participants were students entering ninth or tenth grade. The initial research sample included 1 ,953 
students from 10 sites. Many were from low-income or single-parent households that received 
public assistance or food stamps. Students were randomly assigned to one of three conditions: 

(1) enrollment in Career Academies, (2) waitlist for Career Academies enrollment, and (3) a compari- 
son group. This resulted in a randomized sample of 1 ,064 intervention and 889 comparison students. 
The final analytic sample reflects losses due to (1) individuals who were randomly assigned to 
condition but were subsequently determined to be ineligible for the study, (2) a site dropping out of 
the study, and (3) non-response on the outcomes of interest. No significant differences were found 
between intervention and comparison students at baseline. The size of the analytic samples varied 
for each of the outcomes of interest, ranging from 1 ,379 to 1 ,454 students. Demographic charac- 
teristics for the study (only reported for the 2004 analytic sample) are as follows: 41 .4% were male 
(both intervention and comparison); 56.1% (intervention) and 57.4% (comparison) were Hispanic, 
30.6% (intervention) and 27.8% (comparison) were Black, and 5.7% (intervention) and 7.5% 
(comparison) were White. English language learners accounted for 6.6% (intervention) and 8.9% 
(comparison) of the sample. 


Career Academies Updated September 201 5 


Page 9 




WWC Intervention Report 


Intervention 

group 


Comparison 

group 


The Career Academies included in the study had the following features: 

School-within-a-school organization. A small learning community is formed within the larger 
high school by clustering three to five teachers and 50-75 students per grade. Teachers are 
drawn from academic and career-related disciplines and remain with students from year to year. 
One teacher assumes lead responsibility for administering the academy and serves as liaison 
with school and district administrators and employer partners. Students take two to four courses 
a year in the academy and other courses in the regular high school. Block scheduled academy- 
oriented classes in the morning are followed by regular classes in the afternoon. Academic and 
technical curricula are based on a career theme. Occupational classes are structured around a 
range of areas in a career field. Career themes are chosen based on local employment needs 
and demand for expertise. Among the career themes are health professions, business and 
finance, electronics, travel and tourism, and information technology. 

Focused curricula and enriched learning opportunities. Academies try to bring academic 
rigor to career-related courses and applied learning opportunities to academic courses. 
Academy curricula are intended to ensure that students meet core academic requirements 
for graduation and college preparation and to provide a coherent sequence of technical and 
occupation-related classes. Career awareness and development activities aim to improve 
students’ understanding of the world of work and occupations within the program’s broad 
career theme. Through work-based learning programs developed in collaboration with 
employer partners, students are placed in jobs (or a series of short-term jobs) that expose 
them to occupations. 

Employer partnerships. Relationships with local employers help to support academy programs 
and to provide a basis for work- and career-related activities for students. Employer partners 
contribute funds, and their staff participate as speakers, supervisors of student interns, and 
student mentors. Many academies create advisory boards to guide the development of 
curricular and extracurricular activities. Academy staff and employer representatives develop 
career-awareness and development activities, including field trips, job shadowing, and outside 
speakers from the business community. 

Interpersonal support. Academies function as communities of support for students and 
teachers. Academies attempt to ensure that students get personalized attention from teachers, 
teachers have higher expectations, and students collaborate with peers. Academies try to 
ensure that teachers are supported by opportunities for professional collaboration and by 
adequate resources, and that they have the capacity to influence instructional and administrative 
decisions. Parent involvement is encouraged. 

The comparison group were students randomly assigned not to attend the Career Academies 
program. The majority of these students attended their regular high school and did not enroll 
in any Career Academies classes. Of the students in the comparison group, 7.2% enrolled in 
an academy at some point; 4.3% participated in an academy through graduation. The regular 
high school classes attended by members of the comparison group may have included students 
who were in the Career Academies in their school. Researchers purposefully chose to include 
Career Academies sites where there was a clear contrast between the Career Academies and 
other programs available to potential academy students. 
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Outcomes and 
measurement 


Support for 
implementation 


Eligible outcomes under the WWC’s Dropout Prevention review protocol include the percentage 
of students who graduated or received a GED (in the completing school domain), the percentage 
of students dropping out (in the staying in school domain), and the number of course credits 
completed (in the progressing in school domain). These outcomes were reported in the Kemple 
and Snipes (2000), Kemple (2004), and Kemple and Willner (2008) studies. 

The Kemple and Snipes (2000) study reported on a number of outcomes that are ineligible for 
this review, including math and reading test scores, risky behaviors, steps towards postsecondary 
education and employment, and specific courses taken. The Kemple and Rock (1996) study 
reported a number of teacher outcomes; none of these are eligible. The Kemple (2004) and 
Kemple and Willner (2008) studies reported on employment and earnings post-high school, 
as well as outcomes related to family formation and public assistance, which are not eligible. 
For a more detailed description of these outcome measures, see Appendix B. 

Career Academies teachers came from a variety of academic and vocational disciplines and 
had similar background characteristics as other teachers in the same high school. Professional 
development opportunities included focusing on student-related concerns and on coordinating 
the career development and employer-related activities. All of the schools studied had used the 
Career Academies framework for at least 2 years before the beginning of the study. 
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Appendix B: Outcome measures for each domain 


Completing school 

Earned high school diploma or 
GED 8 years after projected date of 
graduation (%) 

This outcome was measured by the Career Academies post-high school follow-up survey, administered 8 years 
after a student’s projected twelfth-grade year. Students responded whether they had earned a high school 
diploma or a GED certificate (as cited in Kemple, 2008). 

Staying in school 

Dropped out of high school (%) 

This measure counted a student as a dropout at the end of their scheduled twelfth-grade year if they were not 
listed as enrolled on any of three data sources (student survey, district records, and school enrollment status 
reports completed by the host high schools) and if one of the following conditions were met: student reported 
being a dropout on the survey, or school records indicated student had dropped out with no indication of being 
enrolled elsewhere (as cited in Kemple & Snipes, 2000 and through author communication). 

Progressing in school 

Total course credits 

This measure included all course credits students earned from ninth grade through the end of their projected 
twelfth-grade year (until just before they would have graduated from high school). In seven sites, academies 
began in the tenth grade, and ninth-grade course credits were earned before entering the academy. The 
remaining two academies began in the ninth grade, and course credits were earned during 4 years of high 
school (as cited in Kemple & Snipes, 2000). Data on credits were obtained from school records. 
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Appendix C.1: Findings included in the rating for the completing school domain 


Mean 

(standard deviation) WWC calculations 


Outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Kemple & Willner (2008) a 

Earned high school 
diploma or GED 8 years 
after projected date of 

Full 

research 

sample 

1,428 

students 

95.8 

(na) 

93.6 

(na) 

2.2 

0.27 

11 

>.10 


graduation (%) 


Domain average for completing school across all studies 0.27 11 Not 

statistically 

significant 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on outcomes, representing the average change expected for all individuals who are 
given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the change in an 
average individual’s percentile rank that can be expected if the individual is given the intervention. Some statistics may not sum as expected due to rounding, na = not applicable. 
a For Kemple & Willner (2008), no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value presented here was 
reported in the original study. Values shown for the academy group are unadjusted mean values; values shown for the non-academy group are calculated by subtracting the impact 
estimate from the academy group's unadjusted mean values. This study is characterized as having potentially positive effects because the effect is not statistically significant but is 
substantively important. For more information, please refer to the WWC Procedures and Standards Handbook (version 3.0), p. 26. 


Appendix C.2: Findings included in the rating for the staying in school domain 


Mean 

(standard deviation) WWC calculations 


Outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Kemple & Snipes (2000) a 

Dropped out of high school (%) 

Full 

research 

sample 

1,454 

students 

10.1 

(na) 

12.4 

(na) 

2.3 

0.14 

6 

>.10 


Domain average for staying in school across all studies 0.14 6 Not 

statistically 

significant 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. This outcome measures a negative behavior; thus, signs were reversed on the mean difference, effect size, and improvement index to demonstrate that 
the intervention group was favored when negative differences were reported and not favored when positive differences were reported. The effect size is a standardized measure 
of the effect of an intervention on outcomes, representing the average change expected for all individuals who are given the intervention (measured in standard deviations of the 
outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the change in an average individual’s percentile rank that can be expected if 
the individual is given the intervention. Some statistics may not sum as expected due to rounding. na= not applicable. 

a For Kemple & Snipes (2000), no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value presented here was reported 
in the original study. Sample sizes, adjusted means, and standard deviations were provided by the study authors in response to a WWC query. The authors also adjusted the impact esti- 
mates using regression techniques to control for background characteristics and clustering within the program sites. This study is characterized as having an indeterminate effect because 
the effect is neither statistically significant nor substantively important. For more information, please refer to the WWC Procedures and Standards Handbook (version 3.0), p. 26. 
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Appendix C.3: Findings included in the rating for the progressing in school domain 


Mean 

(standard deviation) WWC calculations 



Study 

Sample 

Intervention 

Comparison 

Mean 

Effect 

Improvement 


Outcome measure 

sample 

size 

group 

group 

difference 

size 

index 

p-value 

Kemple & Snipes (2000) a 

Total course credits 

Full 

1,379 

22.3 

21.8 

0.05 

0.09 

4 

.08 


research 

sample 

students 

(5.6) 

(5.5) 






Domain average for staying in school across all studies 0.09 4 Not 

statistically 

significant 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on outcomes, representing the average change expected for all individuals who are 
given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the change in an 
average individual's percentile rank that can be expected if the individual is given the intervention. The statistical significance of the study’s comparisons and domain average was 
taken from the published study. Some statistics may not sum as expected due to rounding. 

a For Kemple & Snipes (2000), no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value presented here was 
reported in the original study. Sample sizes were provided by the authors. This study is characterized as having an indeterminate effect because the effect is neither statistically 
significant nor substantively important. For more information, please refer to the WWC Procedures and Standards Handbook (version 3.0), p. 26. 
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Endnotes 

1 The descriptive information for this program was obtained from publicly available sources: http://www.ncacinc.com/, down- 
loaded November 2014; http://casn.berkeley.edu/, downloaded November 2014; http://americasfutureworkforce.org/2013/10/26/ 
opportunities/, downloaded January 2015; http://www.mdrc.org/project/career-academies-exploring-college-and-career-options- 
ecco#overview/, downloaded January 2015; and Kemple & Snipes (2000). The WWC requests developers to review the program 
description sections for accuracy from their perspective. The program description was provided to the developers in July 2014; 
however, the WWC received no response. Further verification of the accuracy of the descriptive information for this program is 
beyond the scope of this review. 

2 The literature search reflects documents publicly available by December 2014. The previous intervention report was released in 
October 2006. This report has been updated to include reviews of five studies that have been released since 2006 and six studies 
that were released before 2006 and not included in the earlier report. Of the additional studies, seven were not within the scope of 
the review protocol for the Dropout Prevention topic area, and four were within the scope of the review protocol but did not meet 
WWC group design standards. A complete list and disposition of all studies reviewed are provided in the references. 

While the overall rating of the current report did not change from the 2006 report, there are three reasons why some findings differ 
between reports. First, the current report includes findings from a more recent report released by the authors (Kemple and Snipes, 
2008.) Second, the current report excludes an outcome in the progressing in school domain that was included in the 2006 report; the 
Lead Methodologist and Content Expert have now determined that the binary outcome of whether total course credits meets gradu- 
ation requirements is not eligible for review because it is not independent of the total course credits outcome. And third, the current 
report focuses on the full sample of 1 ,890 students, while the 2006 report focused on a high-risk subgroup of 474 students. This 
change was made because the Lead Methodologist and Content Expert have determined that the full sample meets the protocol’s 
criteria for students at risk for dropout, and because the high-risk group no longer meets WWC group design standards. The WWC 
attrition standard changed between version 1.0, used in the 2006 report, and version 3.0, used in the current report. As a result, the 
tenth site, which disbanded during the study and was excluded from attrition calculations in 2006, is now included in the attrition 
calculations. After including this site, the high-risk subgroup is likely to have high attrition and does not demonstrate equivalence 
between the intervention and comparison groups. Although requested in an author query, we were unable to obtain the number of 
youth in each risk group by condition. 

In summary, the effectiveness ratings for the staying in school and progressing in school domains changed from potentially positive 
effects to no discernible effects due to the change in sample and the decreased number of outcome measures included in the progress- 
ing in school domain. The effectiveness rating for completing school changed to potentially positive effects due to the change in the 
focal sample and the inclusion of more recent data reported in Kemple and Snipes (2008). The studies in this report were reviewed 
using the standards from the WWC Procedures and Standards Handbook (version 3.0), along with those described in the Dropout 
Prevention review protocol (version 3.0). The evidence presented in this report is based on available research. Findings and conclusions 
may change as new research becomes available. 

3 For criteria used in the determination of the rating of effectiveness and extent of evidence, see the WWC Rating Criteria on p. 1 6. These 
improvement index numbers show the average and range of individual-level improvement indices for all findings across the studies. 

4 This estimate is derived from the following sources: http://www2.bc.cc.ca.us/techprep/partnership.html and www.ncset.org/publica- 
tions/essentialtools/dropout/part3.3.02.asp 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2015, September). 
Dropout Prevention intervention report: Career Academies. Retrieved from http://whatworks.ed.gov 
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WWC Rating Criteria 

Criteria used to determine the rating of a study 

Study rating 

Criteria 

Meets WWC group design 
standards without reservations 

A study that provides strong evidence for an intervention’s effectiveness, such as a well-implemented RCT. 

Meets WWC group design 

A study that provides weaker evidence for an intervention's effectiveness, such as a QED or an RCT with high 

standards with reservations 

attrition that has established equivalence of the analytic samples. 

Criteria used to determine the rating of effectiveness for an intervention 

Rating of effectiveness 

Criteria 

Positive effects 

Two or more studies show statistically significant positive effects, at least one of which met WWC group design 
standards for a strong design, AND 

No studies show statistically significant or substantively important negative effects. 

Potentially positive effects 

At least one study shows a statistically significant or substantively important positive effect, AND 

No studies show a statistically significant or substantively important negative effect AND fewer or the same number 

of studies show indeterminate effects than show statistically significant or substantively important positive effects. 

Mixed effects 

At least one study shows a statistically significant or substantively important positive effect AND at least one study 
shows a statistically significant or substantively important negative effect, but no more such studies than the number 
showing a statistically significant or substantively important positive effect, OR 

At least one study shows a statistically significant or substantively important effect AND more studies show an 
indeterminate effect than show a statistically significant or substantively important effect. 

Potentially negative effects 

One study shows a statistically significant or substantively important negative effect and no studies show 
a statistically significant or substantively important positive effect, OR 

Two or more studies show statistically significant or substantively important negative effects, at least one study 
shows a statistically significant or substantively important positive effect, and more studies show statistically 
significant or substantively important negative effects than show statistically significant or substantively important 
positive effects. 

Negative effects 

Two or more studies show statistically significant negative effects, at least one of which met WWC group design 
standards for a strong design, AND 

No studies show statistically significant or substantively important positive effects. 

No discernible effects 

None of the studies shows a statistically significant or substantively important effect, either positive or negative. 

Criteria used to determine the extent of evidence for an intervention 

Extent of evidence 

Criteria 

Medium to large 

The domain includes more than one study, AND 
The domain includes more than one school, AND 

The domain findings are based on a total sample size of at least 350 students, OR, assuming 25 students in a class, 
a total of at least 14 classrooms across studies. 

Small 

The domain includes only one study, OR 
The domain includes only one school, OR 

The domain findings are based on a total sample size of fewer than 350 students, AND, assuming 25 students 
in a class, a total of fewer than 14 classrooms across studies. 


Career Academies Updated September 201 5 


Page 16 






WWC Intervention Report 


Glossary of Terms 

Attrition 

Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Extent of evidence 

Improvement index 

Intervention 
Intervention report 


Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Rating of effectiveness 


Single-case design 


Attrition occurs when an outcome variable is not available for all participants initially assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review and inclusion in this report if it falls within the scope of the 
review protocol and uses either an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

An indication of how much evidence supports the findings. The criteria for the extent 
of evidence levels are given in the WWC Rating Criteria on p. 16. 

Along a percentile distribution of individuals, the improvement index represents the gain 
or loss of the average individual due to the intervention. As the average individual starts at 
the 50th percentile, the measure ranges from -50 to +50. 

An educational program, product, practice, or policy aimed at improving student outcomes. 

A summary of the findings of the highest-quality research on a given program, product, 
practice, or policy in education. The WWC searches for all research studies on an interven- 
tion, reviews each against design standards, and summarizes the findings of those that 
meet WWC design standards. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which study participants are 
assigned to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which eligible study participants are 
randomly assigned to intervention and comparison groups. 

The WWC rates the effects of an intervention in each domain based on the quality of the 
research design and the magnitude, statistical significance, and consistency in findings. The 
criteria for the ratings of effectiveness are given in the WWC Rating Criteria on p. 16. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 


Career Academies Updated September 201 5 


Page 17 


WWC Intervention Report 


Glossary of Terms 


Standard deviation The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample tend to be spread out over a large range of values. 

Statistical significance Statistical significance is the probability that the difference between groups is a result of 

chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < .05). 


Substantively important a substantively important finding is one that has an effect size of 0.25 or greater, regardless 

of statistical significance. 

Systematic review a review of existing literature on a topic that is identified and reviewed using explicit methods. 

A WWC systematic review has five steps: 1) developing a review protocol; 2) searching the 
literature; 3) reviewing studies, including screening studies for eligibility, reviewing the meth- 
odological quality of each study, and reporting on high quality studies and their findings; 
4) combining findings within and across studies; and, 5) summarizing the review. 


Please see the WWC Procedures and Standards Handbook (version 3.0) for additional details. 
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Intervention 

Report 



Practice 

Guide 



Quick 

Review 


Single Study 
Review 



An intervention report summarizes the findings of high-quality research on a given program, practice, or policy in 
education. The WWC searches for all research studies on an intervention, reviews each against evidence standards, 
and summarizes the findings of those that meet standards. 


This intervention report was prepared for the WWC by Mathematica Policy Research under contract ED-IES-13-C-0010. 
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